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SUMMARY 

First  this  paper  deals  with  the  problem  of  logical  design  of  an 
optimum  network  such  that  the  network  consists  of  only  building  blocks 
chosen  out  of  a  given  set  of  building  blocks.   A  design  procedure  is 

formulated  based  upon  the  integer  linear  programming  approach  of  the 

[91 
first  paper   .   The  formulation  contains  threshold  logic  expression  as 

an  auxiliary  means,  even  though  the  building  blocks  are  actually  implemented 

with  conventional  gates.   An  extension  of  the  above  procedure  to  the 

synthesis  with  multiple-output  building  blocks  is  easy.   Networks  with 

the  minimum  number  of  blocks  were  actually  designed  for  all  switching 

functions  of  three  variables,  assuming  that  NOR  and  AND  are  only  available 

building  blocks.   The  result  shows  the  computational  feasibility  of  the 

design  procedure. 

Next  logical  design  of  an  optimum  sequential  network  by  the 
integer  linear  programming  approach  is  discussed.   It  has  been  known 
that  a  sequential  network  with  the  minimum  number  of  gates  may  not  be 
realized  if  the  number  of  states  in  the  state  transition  table  is  minimized 
first.   A  procedure  to  design  a  sequential  network  of  the  minimum  number 
of  gates,  no  matter  whether  the  number  of  the  states  is  the  minimum  or 
not,  is  formulated  by  integer  programming  approach.   However  before 
discussing  the  above  procedure,  the  design  procedure  of  a  sequential 
network  with  the  minimum  number  of  states  is  also  formulated  as  an  integer 
program  since  it  can  be  treated  more  easily. 

The  integer  programming  approach  can  be  applied  to  diagnosis 
of  a  network  also. 


1.   Introduction 

Probably  one  of  the  most  important  problems  in  logical  design  is 
the  design  of  an  optimum  network  which  realizes  a  given  function  (or 
functions),  under  the  condition  that  the  network  consists  of  building 
blocks,  each  chosen  from  a  given  set  of  different  types  of  building  blocks. 
(Each  building  block  itself  may  be  a  small  network  of  gates.)  A  simple 
example  is  the  design  of  an  optimal  network  when  AND  and  EXCLUSIVE-OR  are 
given  as  building  blocks.  A  network  parameter  to  be  optimized  varies 
depending  on  engineering  motivation.  Examples  are  the  number  of  building 
blocks,  the  number  of  levels  and  the  number  of  interconnections.  This 
design  problem  is  not  an  easy  one  particularly  when  restrictions  such  as 
the  maximum  fan-ins  or  the  maximum  fan-outs  are  imposed  on  a  network, 
or  when  a  network  has  multiple  outputs. 

When  an  optimum  sequential  network  which  consists  of  building 
blocks  each  chosen  from  the  given  building  blocks  is  to  be  designed,  the 
problem  is  even  more  difficult. 

In  this  paper,  we  will  show  that  these  design  problems  can  be 

formulated  as  integer  linear  programming  problems,  extending  the  integer 

[9l 

linear  programming  approach  of  the  previous  paper, Part  I   .  Most  of  our 

formulations  in  this  paper  are  not  tested  computationally  and  may  be 
simply  of  theoretical  interest  at  this  stage  of  integer  linear  programming 
algorithms.  But  since  the  design  of  an  optimum  network  of  NOR  gates 
and  the  design  of  an  optimum  network  of  NOR  and  AND  gates  (Section  5  of 
this  paper  and  [l])  which  is  a  special  case  of  the  general  formulations  in 
this  paper  are  computationally  feasible,  at  least  some  of  other  formulations 
also  might  be  solvable  in  a  reasonable  computation  time. 

The  integer  linear  programming  approach  can  be  applied  to  diagnosis 
of  a  given  network  also. 
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2.   Integer  Linear  Inequality  Expression  0:  Generative  Building  Block 

First  we  want  to  design  an  optimal  network  which  consists  of 
building  "blocks  each  chosen  .from  a  given  set  of  building  blocks.  Assume 
the  number  of  given  building  blocks  is  B.   Let  g  ,  . . . ,  g  denote  different 
switching  functions  which  the  outputs  of  given  B  building  blocks  respectively 
represent.   Let  the  building  block  g,,  building  block  g_,  . ..,  building  block 

g^  have  p_  input-terminals,  . ..,  p_  input-terminals,  respectively.   Set 

r>  l  a 

H  =  Max  {p_,  . ..,  p_).  (2.0.1) 

By  adding  idle  input  terminals,  we  can  assume  that  every  building  block 

has  H  input -terminals.  H  represents  the  maximum  fan-ins  restriction  to 

each  building  block. 

Before  discussing  the  detailed  integer  linear  programming  formulation, 

let  us  outline  the  whole  design  scheme.  First  we  formulate  a  conceptual 

building  block  called  a  generative  building  block,  which  can  express  any  one 

of  g_ ,  ....  g_  according  to  parameters  associated  with  them.   The  generative 
1       B 

building  blocks  then  constitute  a  feed-forward  network  to  realize  a 
given  function*  f  for  external  input  vectors  x    ,  0=1,  2,  ...,  m. 
This  feed-forward  network  can  also  be  written  in  inequalities  with  integer 
variables  as  will  be  shown  in  the  following  sections.  The  resultant 
integer  linear  program  is  then  solved  by  a  similar  procedure  as  proposed 
in  Part  I  "   and  an  optimum  network  can  be  obtained  in  a  finite  number 


*   The  following  procedure  can  be  extended  to  the  multiple  output 
network  synthesis  by  using  the  same  technique  developed  in 
Section  7  of  Part  I  [9].  Also  see  Remark  1+.2.2. 


of  steps.  An  import ant  difference  from  the  case  of  Part  I  lies  in  that 
in  this  paper  we  consider  a  generative  building  block  instead  of  a 
threshold  gate  of  Part  I  and  determine  the  values  of  parameters  in 
each  generative  building  block,  letting  the  generative  building  block 
represent  only  one  of  g_,  . ..,  g_. 

This  section  deals  with  the  expression  of  a  generative  building 
block  by  integer  linear  inequalities. 
2.1.    Integer  Linear  Inequality  Expression  Of  The  Aggregate  Of 

&2_t    •••>    Sg 

The  first  step  to  derive  a  generative  building  block  is  to 
express  all  given  building  blocks  g_,  ...,    g_  by  integer  linear  inequalities. 
The  selection  of  one  building  block  out  of  these  in  each  generative 
building  block  will  be  discussed  in  Section  2.2. 

Several  methods  have  been  devised  to  express  g_  through  g^ 
Here  a  method  based  on  the  representation  of  g_  through  g^  by  a  network 
of  threshold  gates  will  be  given.     Other  methods  based  on  a  multi- 
threshold  threshold  gate  and  based  on  the  direct  specification  of 
true  and  false  vectors  of  each  of  g.,  . . . ,  g_  by  integer  linear  inequalities 
will  be  given  in  Appendices. 


Fig.  2.1.1  Collective  expression 
of  g  ,  . . . ,  g_by  a  multiple 

output  network  of  threshold  gates. 


Now,  given  functions  g_  through  £_  of  H  variables,  let  us  realize 

J.        a 

these  functions  by  a  multiple  output  network  of  threshold  gates  as  shown 
in  Fig.  2.1.1.   Our  aim  is  to  derive  the  relation  between  H  input  variables 
y, }    •  •  ♦ )   y-rr  a*id  B  output  functions  g_ ,  . . . ,  g_,  in  the  form  of  integer 
linear  inequalities.  Although  any  network  which  realizes  g_,  ....  g_. 

J-  -D 

may  be  used  to  express  a  generative  building  block,  a  network  with  the 
minimum  number  of  threshold  gates  is  preferable  because  it  gives  a 
concise  expression.   Threshold  gates  are  used  sionply  as  a  mathematical 
tool,  no  matter  what  physical  devices  are  employed  to  implement  g_,  ...,  g^. 
The  network  may  be  obtained  by  using  the  integer  linear  programming 

ro] 

approach  described  in  Part  I     without  any  network  restrictions.   Of 
course  any  other  method  known  so  far  can  be  used  for  this  purpose. 
Several  methods  will  be  found  in  [£],  [11]. 

Let  us  number  threshold  gates  from  1  to  C  in  a  synthesized 
feed-forward  network  with  multiple  outputs  in  which  each  gate  receives 
inputs  only  from  the  preceding  gate  and  variables  y_,  ...,  y  . 
Assume  that  the  C  -th  gate  realizes  g  ,  r=l,  2,  ...,  B.  Let  the  output 

value  of  the  k-th  gate  for  the  input  vector  y    =  (y.,   ,  <>..,  y\-  ) 

(j) 
"be  On  )    j=lj  2,  ...,m.   Let  ex..    denote  the  weight  from  the  i-th  gate  to 

&■  lk 

if 
the  k-th  gate,w  denote  the  weight  of  the  variable  y»  to  the 

k-th  gate  and  Tn  denote  the  threshold  of  the  k-th  gate.  Note  that  en.,  , 

°        k  ik7 

w  and  T  are  known  constants,  though  they  were  unknown  variables  in  the 
synthesis  problem  of  Part  I  ^  .  But  y   ,  Q.   and  Qj^  are  unknown  now. 
With  these  notations  defined,  the  k-th  gate*  in  the  network  for 


*     Different  from  the  case  of  Part  I,  even  the  gate  for  an  output  function 

g  needs  two  inequalities  of  (2.1.1), 
r 


each  k  is  expressed  in  inequalities;* 

H   k  (i)   k"1 
Z  v*  y^   +  Z 

i=l  *  *  ±=1 

(2.1.1) 


H  k— 1 


"  £  wi  ^   "  £  "ik  ^  >  -  V  *  "  D  ^ 

(  j  =  1,  2,    ...,  m). 

These  inequalities  are  interpreted  in  the  similar  manner  as  those  of 
Part  I  [9].  Namely,  (2.1.1)  are  converted  into  the  following; 

Z  w*  y^  +  Z  aik  Q^*}  >  Tk     if   q£3)  =  1 

Z  w*  yjd)  +  Z  a.k  Qp}  <  Tk  -  1  if   o£d)  =  0  . 

In  (2.1.1)  y   ,  Q.   and  Q^'  are  only  unknown  variables.  When  the 

network  is  synthesized  such  that  the  C  -th  gate  realizes  g  ,  Q 

r 


together  with  the  relation 


represents  g  (y  ^  ).  The  set  of  inequalities  (2.1.1)  for  k  =  1,  2,  ...,  C 


Q(d)  =  ^  {^)h  (  J  =  1,  2,  ...,  m). 

(r  =  1,  2,  ...,  B$. 

represent  the  functional  relations  between  y  ,  . ..,  r  and  g  ,  . ..,  g^. 

Thus  we  have  formulated  the  aggregate  of  building  blocks  g  ,  . . . ,  g_ 


*  The  constant  U  used  in  inequalities  (2.1.1)  is  a  sufficiently 

large  positive  number  such  that  if  q)   =  1,  for  example,  the 

second  inequality  of  (2.1.1)  becomes  non-restrictive,  and  if 

Q^   =  0,  then  the  first  becomes  non-restrictive.  Although  the 

minimum  magnitude  of  U  which  is  necessary  to  have  this  property 
may  vary  with  each  inequality,  the  same  notation  U  will  be  used 
throughout  this  paper. 


in  integer  linear  inequalities.  This  is  the  first  step  of  our  derivation 
of  a  generative  building  block.  The  second  step  -will  be  discussed  in 
the  next  section. 
Example  2.1.1. 

Let  us  find  an  integer  linear  inequality  expression  of  two 
building  blocks: 

£]_  =  yx  cD  y2 

These  two  functions  g_  and  g^  can  be  realized  by  two  threshold  gates 
connected  as  shown  in  Fig.  2.1.2. 

y.  J-  ,— ,  q[J) 

]V^(T=2  ) — *-  g2  =  yi  y2 

V~  \  I 

y2  . 

yi  ^-r^   2         -    n 

i  AT-1  J   "~^  gl  yl^y2  Fig.  2.1.2  Realization  of  two  building 
y2  ^         -^  blocks  y  ©  yp  and  y  y2  by  a  network 

of  threshold  gates, 
According  to  the  above  discussion,  the  outputs  of  threshold  gates 
Q-.J'  and  tU   for  each  y  ^ J  ,  j  =  1,  2,    . . . ,  m,  can  be  defined  by 
the  following  inequalities: 

y[3)  +  y^3)  >  2  -u  (l-Q^) 
-4d)  -yp^-i-UQ^, 


(  j  =  1,  2,     ... ,    m.) 


y[d)  +yp)  -2  Q^}  >    l  -  u  (l-Q^) 
-y£j)  -y^)+2^')>  0-TJQ^, 

(j  =  1,  2,  ...,  m.). 
Any  value  of  U  >  3  is  sufficent  for  these  inequalities. 
If  we  consider  that 


Sl  <r(3)>-^J) 


(2.1.2) 


% 


G{i)   =43) 


for  each  y    ,  j  =  1,  2,  . ..,  m,  these  inequalities  describe  relations 
between  the  input  values  and  the  output  values  of  g_  and  g  . 

Remark  2.1.1 

When  a  certain  function  g.  and  its  negation  g  are  given  as 
building  blocks,  we  can  eliminate  one  of  them  from  the  formulation  by 
the  relation 

ir  =  i  -  V 

Remark  2.1.2 

In  many  cases,  some  of  g^,  . ..,  g_  are  symmetric  in  some  of 
variables.   In  this  case,  weights  corresponding  to  symmetric  variables 
can  be  equated.  Therefore  the  inequalities  will  become  somewhat  simpler. 


2.2.   Selection  Parameters  Of  Generative  Building  Block 

Since  a  generative  building  block  has  to  express  one  of  g  ,  . ..,  g  , 

parameters  to  select  one  of  the  output  functions  g  ,  . . . ,    g^  which  were 

expressed  in  inequalities  (2.1.1)  must  be  incorporated. 

As  these  selection  parameters. let  us  introduce  variables 

9n.    ....  9^   whose  values  are  1  or  0  such  that  if  6      =  1,  then  a 
If  '      B  r  ' 

generative  building  block  represents  the  corresponding  building  block 

g  .   In  order  to  select  only  one  building  block  out  of  all  B  building 

blocks  g  ,  . . . ,  g_,  the  following  equality  is  added. 

9±   +  62   +  ...  +9=1.  (2.2.1) 

Let  P    denote  the  value  which  the  output  of  the  generative  building 

block  should  assume  for  the  j-th  input  vector  y   .   In  other  words, 

P  J  satisfied  the  relation 

p(J)  =  gr  (y(j))  =  Q^}  (2.2.2) 

r 

if  6     =   1  holds,  where  Q,  J  is  the  output  value  of  the  threshold  gate 

r 

which  realizes  g  as  discussed  in  Section  2.1.   This  is  expressed  in 


inequalities; 


p(j)  ,:  QU)  +  (l_e  ) 

—  c        r 

r 

p(j)  >Q(j)  +  (1_9  ) 

—  c        r 

r 


(2.2.3) 


(r  =  1,  2,    ...,    B) 
(j  =  1,  2,    ...,   m). 


Obviously,  if  9     -   0  then  the  inequalities  become  non-restrictive 

and  if  9     =   1  the  inequalities  force  P    to  be  equal  to  Q   . 

r 
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Consequently,  with  inequalities  (2.1.1),  (2.2.1)  and  (2.2.3) 

together,  a  generative  building  block  which  is  able  to  choose  one  of 

g_ ,  ...,  g-n  is  completely  described.  This  expression  will  be  the  basis 
l      a 

for  the  optimum  network  synthesis  procedure  in  the  following  sections. 
Remark  2.2. 1 

Throughout  this  paper  only  a  generative  building  block  which  has 
a  single  output  is  dealt  with.  However  a  generative  building  block  with 
more  than  one  output  is  also  important  for  the  engineering  practice. 
For  example  in  some  physical  realization  of  an  OR  gate,  NOR  is  also 
available  on  the  same  gate  without  extra  hardware  except  a  lead  wire. 
The  generalization  of  our  integer  linear  programming  approach  to  the 
design  of  a  network  with  building  blocks  which  have  multiple  outputs  is 
straight  forward  and  will  not  be  discussed  in  this  paper. 
Remark  2.2.2 

When  a  certain  type  of  device  is  employed  to  implement  building 
blocks,  Wired- OR  is  easily  available  by  connecting  the  outputs  of  other 
gates  without  extra  device.  Wired-OR  can  be  interpreted  as  one  of  the 
building  blocks.  Since  only  outputs  of  other  building  blocks  can  usually 
be  connected  to  Wired-OR,  restrictive  inequalities  must  be  added.  Also 
inexpensiveness  of  the  Wired-OR  must  be  taken  into  account. 


3.  Description  Of  A  Feed-Forward  Network  Consisting  Of  Generative 
Building  Blocks 

In  this  section  we  will  describe  a  feed-forward  network  of 
generative  building  blocks  in  linear  inequalities  of  integral  variables. 
3.1.   General  Case 

Consider  a  feed-forward  network  consisting  of  R  generative  building 
blocks,  as  shown  in  Fig.  3«1» 
Assume  that  the  output  of  a 


x 


Fig.  3»1»1  Feed-Forward  Network 
Of  R  Generative 
Building  Blocks 


generative  building  block 
is  not  symmetric  in 
variables*  The  input 
terminals  of  each  generative 
building  block  must  be 
numbered  for  identification. 

Let  cp.(d,k)  denote  a  (0,l)-  variable  which  expresses  the  inter- 
connections from  the  output  of  the  d  th  generative  building  block  to  the 
i  th  input  terminal  of  the  k  th  building  block.   If  cp.(d,k)  =  0,  it  means 
no  interconnection  and  if  cp.(d,k)  =  1,  the  interconnection  exists. 
However  note  that  there  may  be  more  than  one  interconnection  from  the 
d  th  building  block  to  the  k  th  building  block. **  In  other  words,  we 

may  have  cp.  (d,k)  =  cp.  (d,k)  =  ...  =1  for  the  same  d  and  k.   Let  v.(i,k) 

xl         x2  x 

denote  a  (0, l)-  variable  which  expresses  the  connection  of  an  external 

variable  x,  to  the  i  th  input  terminal  of  the  k  th  generative  building  block. 


*     The  case  where  the  output  of  a  generative  building  block  is 
symmetric  in  all  variables  will  be  discussed  in  Section  3*2.   The 
intermediate  case  where  the  output  is  partially  symmetric  i.e. 
symmetric  in  each  of  sets  of  variables  will  not  be  discussed  but 
can  similarly  be  treated. 

*-*  A  degenerate  function  such  as  x_  V  x_  by  substituting  x  into 

x~  of  xx  v  x_  is  generated. 
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Here 

d  =  1,  2,  . ..,  k-1 
1  =  1.  2 ,  •  *  •  ,  H 

v.(+,k)  is  the  connection  from  the  constant  input  of  the  value  1  and 

v.(-,k),  the  connection  from  the  constant  input  of  the  value  0. 

Each  input  terminal  receives  exactly  one  connection,  i.e.  for 

each  k  and  each  i 

k-1  n,+  - 

S  cp.(d,k)  +   S    v.  (i,k)  =  1.             (3-1.1) 

d=l  1        i=l    1 

The  input  value  y.   (k)  at  the  i  th  input  terminal  of  the  k  th  generative 

building  "block  for  the  j  th  external  input  vector  x  J  satisfies  the  following 
condition, 

f  P(j)  (d)    if  q>  (d,  k)  =  1 

y<%)=   (i) 

[  xyJ         if  v.u,  k)  =  i,  (3.1.2) 

(i  =  1,  2,  ...,  H) 
(k  =  1,  2,  . ..,  R). 

where  P   (d)  is  the  output  of  the  d  th  generative  building  block  for  the 
j  th  external  input  vector  x  J  and 


x™'   =   1  and  x™)   =  0  u  (3-1.3) 

(3*1,2)  is  expressed  in  the  following  set  of  linear  inequalities  of  integral 
variables: 

11 


yp^k)  <P(d)(d)  +  (1  -  cp.(d,k)) 

P(d)(d)  <  yp}(k)  +  (1  -  cp.(d,k)),  (3.1.U) 

(k  =  1,   2,  ...,  R) 

(a  =  1,  2,  ...,  k-l) 

(i  =  1,   2,  ...,  H) 

(j  =  1,   2,  ...,  m), 


and 


yp}(k)  <x{p  +■  (l  -v.    U,    k)) 


Xt 


(J)      ^Jd) 


<y^j;  (k)  +  (l  -  v±  (i,  k)),  (3.1-5) 

(k  =  1,   2,    ...,   R) 

(i  =  1,   2,    ,..,    n,    f,    -) 

(i   =   1,    2,    ...,    H) 
(j   =  1,    2,    . ..,   m). 

cp.(d,   k)   =  0  makes    (3»1«^)  non-restrictive  and  cp.(d,   k)   =  1  yields 

y.J"'(k)   =  P^'^(d),    and  so  forth. 

Note  that  the  external  variables  xy     are  given  in  advance  as 
input  vectors  to  the  whole  network  and  considered  as  constants  in  our 
integer  linear  programming  approach.  The  outputs  of  the  R  th  "building 
"block  are  also  specified  in  advance  as  the  values  of  the  given  function 
f(x    )  for  all  j's.   Therefore  P   (r)   are  also  given  constants  "but 
P   (k)   for  k  f  R  are  unknown  variables. 
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3.2  Case  When  Generative  Build-Jig  Blocks  Are  Symmetric  In  All  Inputs. 

If  each  generative  "building  block  is  symmetric  in  all  inputs,  in 
other  words,  if  the  input  terminals  of  each  building  block  are  inter- 
changeable, then  each  input  terminal  need  not  be  differentiated  from 
others.   Let  us  sum  up  the  input  values  at  the  k  th  building  block  and 
denote  it  with 


y 


(%)  =  y[J^k)  +  ... +y£3\&).         (3.2.1) 


In  the  following,  only  y   (k)  will  be  used,  eliminating  the  use  of  y.  (k)  fs. 

The  value  which  y   (k)  assumes  is  not  limited  to  0  and  1.  Also  the 

subscript  i  may  be  dropped  from  cp.(d,k)  and  v.  (i,k).  cp(d,k)  and 

v  (i, k)  are  variables  which  represent  the  interconnection  from  the 

d  th  building  block  to  the  k  th  building  block  and  the  connection  from 

the  £   th  external  variable  to  the  k  th  building  block,  respectively. 

However  note  that  in  the  current  case  cp(d,k)  and  v(i,k)  also  are  not 

restricted  to  the  value,  1  or  0,  because  multiple  interconnections,  in 

other  words  the  simultaneous  interconnections  from  an  output  of  a  building 

block  to  more  than  one  input  terminal  of  a  certain  building  block  may 

yield  a  better  network.  However  the  total  number  of  inputs  to  each 

building  block  must  equal  H,  i.e. 

k-1  n,  +,  - 

E  cp  (d,k)  +   Z    v  U,k)  =  H.  (3.2.2) 

d=l  i=l 

y   (k)  is  expressed  as  follows.  Let  us  introduce  new  variables 
p^(d,k)  and  p^'(i,k)  which  satisfy 
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(j)(d,   k)  <  cp(d,   k)       +  U  (1  -  P(j)(d)) 


(J) 


<P(d,   k)         <  pvj;(d,k)    +U(l-Pu;(d))       y 


(3) 


(3.2.3) 


p(d)(d,    k)   <U  P(d)(d), 


J 


(k  =  1,    2,    ...,    R) 
(d  =  1,  2,   ...,   k-l) 
(j  =  1,   2,    ...,  m), 


(J) 


pVd'(J,   k)   <  v   U,    k)   +U  (1  -  x/') 


(dX 


V 


(i,   k)         <P(j)(i,k)  +U  (1  -  x(A 


(3-2.10 


Li 


P^U,   k)  <  U  x^, 


•> 


(i  =  1,  2,    ...,   n,+  ,    -) 
(j  =  1,  2,   „..,  m) 

(k  =  1,   2,    .0.,   R). 


These  inequalities   of   (3-2.3)    and  (3.2.U)    imply 


P(d)(d,   k)   -  cp   (d,   k)   P(d)(d) 


P(d)U,   k)   =  v  U,   k)  x^. 


(3.2.5) 
(3.2.6) 


Since  the  input  of  the  k  th  building  block  is  the  sum  of  contributions 
from  the  outputs  of  other  building  blocks  and  the  external  variables, 


y 


<Hk)  of  (3.2.1)  which  consist 


s  of  y)3'   of  (3.1.2)  is  obtained  as 


11+ 


y(d)(k)  =  S  <p(d,  k)  P(d)(d)  +  S  '  v(i,  k)  x*J) 
d=l  ,0=1         ^ 

=  s  Pu;(a,  k)  +  s  pu;u  ^, 

d=i         i=i  (3.2.7) 


using  (3*2.5)  and  (3.2.6).   Consequently  the  interconnections  in  a  network 

are  described  by  the  linear  inequalities,  (3.2.2),  (3.2.3),  (2.2.U),  (3.2.7), 

together. 

Remark  3»2.1:  The  connection  of  a  constant  input,  1  or  0,  sometimes  yields 

a  meaningless  result.  For  example,  when  an  AND  gate  is  considered,  the 

connection  of  the  constant  input  0  means  that  the  output  of  the  gate 

is  always  0.  This  can  be  avoided  by  adding  extra  constraints,  whenever 

necessary. 
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k.      Procedures  To  Design  An  Optimal  Feed-Forward  Network 

We  are  now  ready  to  discuss  design  procedures  of  an  optimal 
feed-forward  network,  with  all  the  gates  and  the  network  described  in 
inequalities.  The  behavior  of  generative  building  blocks  which  can 
express  any  of  the  given  building  blocks  gi,  ...,  gp  was  expressed  in 
inequalities  in  Section  2,  and  the  interconnections  from  other  generative 
building  blocks  and  the  external  variables  x  in  inequalities  in  Section  3.  The 
set  of  all  these  inequalities  together  with  x  ^'  and  f(x    ),  i.e. 
P^'(R)  (These  x  ^  's  and  f(x  ^ys  are  regarded  as  given  constants 
in  our  set  of  inequalities)  specifics  the  condition  which  the  feed- 
forward network  must  satisfy.  Any  solution  of  this  set  of  linear 
inequalities  realizes  the  given  function  f.  However  if  a  certain  objective 
function  is  introduced  and  optimized  (minimized  or  maximized),  then  an 
integer  linear  programming  problem  has  been  formulated.   If  a  parameter 
of  the  network  such  as  the  number  of  interconnections  can  be  expressed 
as  a  linear  function  and  chosen  as  an  objective  function,  then  an  optimum 
network  with  the  minimum  number  of  interconnections  will  bs  obtained. 

Assume  that  the  feed-forward  network  for  f  has  R  generative 
building  blocks.   Only  one  of  g  '  s  is  chosen  in  each  generative  building 
block  in  an  optimum  solution  of  our  integer  linear  program.  Let  J  and  K 
be  the  numbers  of  the  variables  and  the  inequalities,  respectively,  which 
are  necessary  in  obtaining  the  relations  between  the  inputs  y:  (k)  ,  ..,, 

yi  (k)  and  the  outputs  P   (k)  of  the  k  th  generative  building  block. 
The  set  of  these  inequalities  necessary  for  all  R  generative  building 
blocks  will  be  referred  to  as  E.  Then  the  numbers  of  the  variables 
in  the  entire  formulation  for  this  network  are: 
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m  H  R  for   y.  (k)     *S 


1 


m(R-l)  for   P^\k) 


^(R-l)  for   cpi(d,  k)    »S 


H  R  (n-42)  for   v.  (i,  k)    !s 


and  the  total  number  of  the  variables  is 


RJ  +mHR  +  m  (R-l)  +  2*fcl)  +  HR(n42).  (U.O.l) 


Note  that  the  number  of  P  J  (k)  is  m(R-l)  because 


P(j*\r)  =  f  (x  (d))  (U.0.2) 


and  therefore  P   (R)  are  not  counted.  The  number  of  the  inequalities 
are 

2  HE  for       (3.1-1)* 

m  H  R(R-l)     for       (3-1.1*) 
2  m  H  R(n+2)   for       (3.1-5) 

The  total  number  of  the  inequalities  is 

RK  +  2  HR  +mHR(R-l)  +  2  m.HR(n-f2).  (^-0.3) 

As  discussed  in  Section  3-2,  if  the  building  blocks  are  symmetric 
in  variables,  the  numbers  of  variables  and  inequalities  can  be  fewer  than 
those  given  above. 


*  An  equality  is  expressed  with  two  inequalities 
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k.l     Procedure  1. 

To  obtain  a  feed-forward  network  with  R  building  blocks  which 
realizes  a  given  function  f,  we  need  solve  the  inequalities  E  and  also 
(3-1.1),  (3.1.fc)  and  (3.1.5)  under  the  condition  P^  (R)  =  f  (x  ^  ). 

R   H    k-1  n 

Z   Z  (  Z  cp.  (d,  k)  +  Z  v.  (£,   k))  (l+.l.l) 

k=l  i=l   d=l  x         £=1     x 

may  be  chosen  as  an  objective  function.  This  implies  the  minimization 
of  the  number  of  interconnections  between  building  blocks  and  connections  of 
external  variables  excluding  those  of  the  constants  in  the  network.   Other 
network  parameters  may  be  chosen  instead  as  an  objective  function. 

A  procedure  for  obtaining  a  feed-forward  network  with  the  minimum 
number  of  building  blocks  and  then, as  the  secondary  objective,  the 
minimum  number  of  interconnections  and  connections  (i.e.  a  network  with 
the  minimum  number  of  interconnections  and  connections  among  networks 
with  the  minimum  number  of  building  blocks)  is  as  follows: 

(1)  Set  R  =  1.   (or  set  R  to  a  lower  bound  of  value  necessary 
to  realize  f  if  it  can  be  found  by  certain  means. ) 

(2)  Solve  the  set  E  of  the  inequalities,  (3.1.  l),  (3.I.I1)  and 
(3.1.5),  for  the  network  of  R  generative  building  blocks, 
minimizing  the  objective  function  (U.l.l).   If  there  is  a 
solution,  an  optimum  solution  of  this  problem  gives  an 
optimal  network  for  f.   If  there  is  no  solution,  go  to  (3). 

(3)  Increase  R  by  one  and  go  to  (2). 

This  procedure  will  terminate  in  a  finite  number  of  steps,  if 

the  given  set  of  building  blocks,  g^  ...,  gg  are  complete  (If  any  function 

can  be  realized  by  the  set  of  g  ,  . . . ,  gg,  the  set  is  called  complete. 
See  [6].  ) 
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k.2     Procedure  2. 

The  second  procedure  is  simply  to  solve  the  integer  linear 
programming  problem  with  R  which  is  sufficiently  large  to  realize  f. 
So  the  gradual  increase  of  R  in  Section  k.l   is  not  necessary. 
The  value  of  R  may  not  be  known  in  general.  But  in  some 
cases  fairly  tight  bounds  may  be  available.   The  numbers  of  the  variables 
and  the  inequalities  may  be  much  greater  than  those  in  Section  k.l.      The 
formulation  is  exactly  the  same  as  before  except  (2.2.1)  and  (2.2.3),  because 
(2.2.3)  assumes  that  exactly  one  of  g  ,  . ..,  g^  is  selected  in  each  generative 
building  block  but  in  the  procedure  some  generative  building  blocks  may 
be  discarded  as  unnecessary  ones  in  the  procedure.  Taking  into  account  this 
situation,  let  us  use  the  following,  instead  of  (2.2.1)  and  (2.2.3) 


^  +  e2  +  ...  +  eB  <  1 


(U.2.1) 


p(j)  <  Q(J)  +  d  .  0  ) 


—  c 


p(j)    >Q(J)     _    (1    _    Q  ) 

_^c     ^     r/ 


m   /  .\      B 

Z  PU;  <  m  S 

3=1         r<L 


(r 
(3 


1,    2, 
1,  2, 


(k.2.2) 


..,    B) 
..,  m.  ) 


The  last  inequality  in  (U.2.2)  implies  the  restriction  that  if  all  9     are  0  (the 
generative  building  block  is  discarded),  the  output  of  this  block  is 
always  0.   Then  the  number  of  building  blocks  which  appear  in  an  optimal 
network  is  written  as 

R   B 

E   Z  9      (k)  . 
v 
k=l  r=l 
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where  9   (k)  is  the  selection  parameter  of  the  k  th  generative  building 

"block.   In  order  to  discard  unnecessary  generative  building  blocks,  the 

objective  function 

R    B         R   H   k-1  n 

V  Z    Z     6   (k)+  Z       Z     (  Z     cp.(d,k)  +Z     v.(i,k))    (U.2.3) 

k=l  r=l  r     k=l  i=l  d=l  X  £=1     x 

is  chosen,  where  V  is  a  large  positive  number  such  that 

V  >  max  Z  Z   (  Z   cp.(d,k)  +  Z   v.(  i,k)),  If  V  is  sufficiently  large,  it 
kid1       I     1 

minimizes  the  first  sum  first  and  subsequently  minimizes  the  second  sum. 
(This  is  possible  because  all  the  variables  in  (U.2.3)  are  integral 
variables. ) 

A  merit  of  Procedure  2  is  that  any  network  parameters  can  be  used 
regardless  of  the  minimality  of  the  number  of  building  blocks  if  they 
are  written  in  linear  form.  For  example,  the  number  of  interconnections 
can  be  minimized,  without  minimizing  the  number  of  building  blocks. 

The  second  procedure  is 

(1)  Find  R  which  is  sufficiently  large  to  realize  a  given  function  f. 

(2)  Solve  the  set  of  inequalities,  E  (for  this  E,  (U.2.1)  and 
(U.2.2)  are  used  instead  of  (2.2.1)  and  (2.2.3)),  (3.1.1), 
(3.1-U)  and  (3-1.5),  minimizing  (1+.2.3). 

Remark  U.2.1:   Procedure  2  is  convenient,  if  given  building  blocks, 

g  ,  . . . ,  g^,  have  different  costs  and  the  total  cost  of  the  network  is 

to  be  minimized.  Let  c  be  the  cost  of  g  .   (r=l,  2,  ...,  B).  If  9   (k)  =  1 

is  chosen  for  the  k  th  generative  building  block,  the  cost  of  the  k  th 

generative  building  block  is  c  .   Thus 

R   B 

Z       Z     c  9    (k)  (k.2,k) 

r  r 
k=l  r=l 
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is  the  total  cost  of  the  "building  blocks  throughout  the  network.  This 
could  be  used  as  an  objective  function. 

Remark  U.gL.2:   The  integer  linear  programming  approach  which  has  been 
discussed  in  these  sections  can  be  easily  extended  to  the  design  of  a 
multiple-output  feed-forward  network.  The  extension  is  based  on  the 

technique  of  designing  a  multiple -output  network  discussed  in  Section 

[9] 
7  of  Part  I    .   Suppose  that  S  functions,  f  ,  f ,    . ..,  f  are  given 

and  are  to  be  realized  by  a  network  with  the  minimum  number  of  building 

blocks.   Let  ¥  ,  denote  a  variable  which  represents  the  selection  of  the  output 

P   (k)  of  the  k  th  generative  building  block  as  f ,  and  whose  value  is  1  or  0. 
Y  =1  means  that  the  output  P^'(k)  is  selected  as  f,  for  all  j's.  Then 


R 

k=l  ** 


(t  =  1,  2,  ...,  S).   (U.2.5) 


ft(x  (J))  >  P(j)(k)  -  (1  -  ¥kt)  (k.2.6) 


ft(x(^)  <  P(;3)(k)  +  (1  -  ¥kt), 

(J  =  1>  2,  . ..,  m)  . 
Tnese  inequalities  (U.2.5)  and  {k.2.6)   together  with  the  entire  set  of 
the  inequalities,  E,  (3.1.1),  (3.1.*0,  (3.1.5)  constitute  an  inequalities 
of  our  integer  linear  programming  approach  for  the  synthesis  of  a  multiple- 
output  network.  In  this  case,  P(^(R)  is  also  considered  as  an  unknown 
variable,  because  it  is  not  known  which  of  f^P^OO  actually  represents. 
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This  approach  will  be  used  in  Section  6  for  the  design  of  a  sequential 
network. 

Remark  h.2.  3  The  design  ©f  a  network  for  a  given  function  which  would 
require  a  large  number  of  building  blocks  is  difficult  for  our  integer 
linear  programming  approach.  However  the  following  approach  could  be 
taken.  First  design  a  presumably  non-optimum  network  for  a  given  function 
by  some  means  which  has  been  known.   (Minimization  of  prime  implicants, 
some  heuristic  methods,  or  others. )  Then  select  a  small  part  of  the 
designed  network,  i.e.  a  small  multiple -output  subnetwork,  and  solve  a 
integer  linear  program  for  this.   If  we  repeat  this  procedure  throughout 
the  network  designed,  we  will  have  an  improved  network  for  the  given  function. 
Remark  lug.lj-  Assuming  that  the  minimum  number  of  building  blocks  which  is 
necessary  for  a  network  to  realize  a  given  function  is  known,  we  want 
to  obtain  a  network  with  the  minimum  number  of  different  types  of  building 
blocks,  without  increasing  the  number  of  building  blocks  in  the  network. 
This  objective  can  be  incorporated  by  minimizing  an  objective  function  as 
will  be  discussed  below.   Let  us  consider  a  network  consisting  of  R 
building  blocks  and  introduce  new  (0, l)-  variables  t  ,  r  =  1,  2,  ...,  B, 


which  are  characterized  by: 


k=l   r 


Then  consider  the  objective  function 

B 
S 

r=l 


R 

Z   0  (k)  <  U  t  ,  (U.2.7) 


(r  =  1,  2,  ...,  B.) 


2  Tr  •  (U.2.8) 
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If  9   (k)  assumes  1  for  some  k  (  g  is  selected  for  the  k  th  generative 

building  block  in  the  network),  then  t  assumes  1.  Since  (U.2.8)  shows 
the  number  of  types  of  building  blocks  actually  used,  (k.2.8)   is  to 
be  minimized  for  the  present  purpose. 
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5.   Example  And  Computational  Result. 

As  an  example  of  network  consisting  of  more  than  one  type  of 

"building  block,  we  synthesized  network  with  NOR  gates  and  AND  gates. 

The  networks  of  the  minimum  number  of  gates  and  then,  as  the  secondary 

objective,  of  the  minimum  number  of  interconnections  and  connections  were 

synthesized  for  all  80  functions  of  three  variables  in  5I4.  minutes  on  the 

IBM  360/751  computer.  We  did  not  try  to  speed  up  the  program  and  the 

computation  time  could  be  further  reduced  by  improving  the  program.   The 

result  is  encouraging  and  may  indicate  the  computational  feasibility 

of  our  approach. 

Fig.  5.1  Expression  of  NOR  and  AND. 


First  of  all,  let  us  consider  the  linear  inequality  expression 
for  a  generative  building  block  which  can  express  either  a  NOR  gate  or 
an  AND  gate  (Sections  2).   In  this  formulation,  however,  we  do  not 
impose  the  fan-in  restriction*  on  each  building  block. 

Now  consider  an  expression  for  NOR  gate  in  the  k  th  generative 
building  block  of  the  network.  Obviously,  the  relation  between  inputs 
y   ,  r  =  1,  2,  ...,  H  (=  (k-l)+  n),and  its  output  Qj   is  written  as 


*  The  k  th  generative  building  block  is  assumed  to  have  exactly 
H  (=(k-l)+n)  input  terminals  for  each  k.   If  a  building  block 
turns  out  to  have  actually  fewer  input  terminal  connections, 
connect  constants  of  appropriate  values  to  the  remaining  input 
terminals. 

2k 


follows, as  illustrated  in  Fig.  5«1: 

-   2  y^j)  >  0  -  U  (1  -  Q^h 
r=l 

(5.1) 

E  y^  >  1  -  U  q^\ 
r=l  r   "  1 

(j  =  1,  2,  . ..,  m  ). 

Taking  advantage  of  the  property  that  a  NOR  gate  is  synmetric  in  all 

variables,  we  can  group  all  the  inputs  into  two  sets;  one  is  to  receive 

the  external  variables  and  the  other  is  to  receive  the  outputs  from  the 

preceding  building  blocks  (Section  3«2),.    (5»l)  is  then  rewritten  as 


-  E  v(i,k)  x^}  -   E  cp  (d,k)  P(j)(d)  >  0  -  U  (1-Qp}) 


i=l        "     d=l 


n         /  .\    k-1  /  .\  /  .\ 

E  vU,k)  x/;  +   E   cp  (d,k)  PU;(d)  >  1  -  U  Qu;, 
i=l        ^      d=l  X 


(j  =  1,  2,  ...,  m), 


(5.2) 


where  E  v(i,k)  x^  +  E  cp  (d,k)  P^'(d)  denotes  the  sum  of  contributions 

from  the  external  variables  and  the  outputs  of  the  other  building  blocks. 
However,  these  inequalities  are  not  linear  because  cp(d,k)  and  P^(d)  are 

both  unknown  variables.  Therefore  new  variables  p    (d,k),  such  that 

P(j)(d,k)  =  cp  (d,k)  P(j)(d)  (5.3) 

are  introduced  (see  (3.2-5)  and  (3.2.7))  and  (5.2)  becomes  the  new  set  of 
linear  inequalities : 


25 


n  /  .n   k-1   /  .\ 


■^T^T"  £  P   (d'k)   >  0  -  Ud-Q^) 


n  /  .n   k-1   /  .\  /  .% 

Z  v  (i,k)  x/;+   Z   pu;(d,k)  >  1  -  U  Qu;, 
i=l        ^    d=l  x 

(5.U) 
(j  =  1,  2,  ■>..,  m). 

v  (i,k)  x,^  is  not  substituted  by  p^'(^,k)  as  in  (3.2.6)  because  this 
is  already  linear  (x\   s  are  given  constants).  To  express  the  relation 

(5.3)  in  linear  form,  the  following  is  used  instead  of  (3.2.3)  in  order  to 
reduce  the  number  of  inequalities. 


-  P^(d)  -  cp  (d,k)  +  p^'(d,k)  +  1  ;  0 


(5.5) 


P^'(d)  +  cp  (d,k)  -  2  p^(d,k)  >  0 

This  is  possible  because  p^'(d,k)  is  known  to  be  0  or  1  in  this  case. 
In  a  similar  way,  an  AND  gate  of  the  k  th  generative  building 
block  is  written  as  follows: 


n  /  .  n    k-1   /  .  %  n  k-1 

Z  v(i,k)  x\3)   +   Z   pU;(d,k)  >   Z  v(i,k)    +   Z  cp  (d,k) 
Z=l  d=l  i=l  d=l 


-  U 


(l-^d)) 
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n  /  .n   k-1   ,  .n  n  k-1 

-  Z   v(i,k)  x^;  -  Z   pu;(d,k)  >  -   Z  v(i,k)  -   Z   cp(d,k) 
i=l         ^    3=1  jg=l  d=l 

+  1  -  U  Q^, 

(5.6) 
(j  =  1,  2,  ...,  m), 

where  Q^J  is  the  output  value  of  the  AND  gate. 

Combining  (5.U)  and  (5.6),  and  introducing  a  selection  variable 
0(k)  (see  Section  2.2),  the  k  th  generative  building  block  is  described 
by 


Z  v(i,k)  x^  -   Z   p^(d,k)     0  -  U  (1-  P^'(k))  -  U  (l-  9   (k)) 
i=l        l  d=l 


Z  vU,k)  x[^   +   Z   p(^(d,k)  >  1  -  U  P(^(k)  -  U  (1-  0(k)) 
£=1  l  d=l 


n         /  .s   k-1   /  .>>  n  k-1 

Z  v(4,k)  x)3)   +  Z   pu;(d,k)  >   Z  v(i,k)  +  Z    cp(d,k) 
i=l        ^     d=l  £=1  d=l 


-  U  (1-  P(j)(k))  -  U  0(k) 


n  /  .\   k-1   /  .n  n  k-1 

Z  vU  ,k)  x^;-   Z   pU;(d,k)  >  -   Z  v(i  ,k)  -   Z  cp(d,k) 
4=1         ^     d=l  £=1  d=l 

+  1  -  U  P^'(k)   -  U  e  (k), 

(5.7) 
(k  =  1,  2,    ...,  R-l) 

( j  =1,  2,  . . .,  m), 
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where  P   (k)  is  the  output  value  of  the  k  th  generative  building  block 
for  the  j  th  external  input  vector.   0=1  means  that  NOR  is  selected  and 
0=0  means  that  AND  is  selected.  For  the  last  generative  building  block, 


n         /  .>,    R-l  /  .n 

Z  vU,R)  xy}    -   Z  plj;(d,R)  >  0  -  U  (1-0  (R)) 
i=l        l  d=l 


(5.8) 


n         /  .  n    R-l  / .  \  n  R-l 

Z  v(i,R)  x\d)   +   Z  pu;(d,R)  >  S  v(i,R)  +  Z  cp(d,R)  -  U  0  (R) 
i=l  d=l  1=1  d=l 


or 


for  f(x  («")  =  1 


n         /  .  \    R-l  / . n 

Z  v(-M)  x-;   +   S  PU;(d,R)  >  1  -  U  (1-0(R)) 
i=l        *      d^J. 


(5-9) 


n         /  .v    R-l  /  .n  n  R-l       \ 

-   Z  v(i,R)  xy}    -   Z  pU;(d,R)  >  -   Z  v(i,R)   -   Z  cp(d,R)+  1  -  U0  (R) 
1=1  l  d=l  1=1  d=l 

for  f  (x  ^')   =   0, 

(j  =  1,  2,   ...,  m). 

Also,  to  describe  the  non-linear  relation  p^(d,k)  =  P^(d)  cp  (d,k) 
in  linear  form,   (5.  5)  is  necessary  for 

K.   =  JLy       <—  y        •  •  •  y       X\ 

d  =  1,  2,    ...,  k-1 

j  =  1,  2,  . . .,  m. 

Consequently  (5-5)*  (5«7),  (5.8)  and  (5-9)  together  completely  represent 
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a  network  consisting  of  NOR  and  AND  gates. 

The  formulation,  in  the  previous  sections  however,  is  simplified 
in  several  respects  by  considering  particular  properties  of  NOR-AND 
network. 

For  example,  selection  variables  6   (k)  and  0p(k)  respectively 
for  NOR  and  AND,  are  reduced  to  a  single  variable  9   (k)  by  using 

01(k)  +  02(k)  =  1,  (5.10) 

of  (2.2.1).   Variables  Q^"  and  Q^*3'  for  the  k  th  generative  building 
block  are  also  replaced  by  a  single  variable  P   (k)  for  the  following 
reason.   If  0(k)  =  1,  say  (i.e.  NOR  is  represented  by  a  generative  building 
block),  only  inequalities  for  a  NOR  gate  (the  first  two  of  (5«7))  becomes 
restrictive  and  others  for  an  AND  gate  becomes  non-restrictive.  Therefore, 
even  if  the  restriction  Qjj'  =  Q^J  =  P^(k)  is  incorporated,  it  does 
not  change  the  nature  of  the  inequalities. 

As  explained  in  the  paragraph  after  (5»h),   p   (i,k)  for  external 
variables  are  also  deleted.  A  significant  reduction  of  the  number  of 
variables  y^(k)'s  is  obtained  by  substituting  y^'(k)  by  (3.2.7). 

In  most  cases,  if  types  of  building  blocks  are  concretely  given, 
we  can  reduce  the  size  of  problem,  by  considering  special  properties  of 
the  types.  However,  we  will  not  try  a  detailed  discussion  about  these 
possibilities  here. 

With  all  these  inequalities,  all  the  optimum  networks  for  each 
three  variable  switching  function  which  have  the  minimum  number  of  building 
blocks  and,  secondary,  the  minimum  number  of  interconnections  (also  connections) 
were  obtained.   The  integer  programming  algorithm  used  for  this  computation 
is  the  implicit  enumeration  method  with  modification        programmed 
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on  the  IBM  360/75I  with  FORTRAN  IV.  A  derailed  description  of  the  algorithm 
will  he  found  in  the  reference  [5].  To  speed  up  the  computation,  further 
properties  of  networks  were  taken  into  account  and  a  number  of  additional 
constraints  were  incorporated  either  to  eliminate  unnecessary  inter- 
connections in  a  network,  or  to  preclude  solutions  which  can  he  obtained 
from  other  solutions  by  permuting  variables.  These  additional  inequalities 
are  vitally  important  for  reducing  computation  time.   This  is  discussed 
elsewhere  as  well  as  detailed  statistics  of  computation. 

Table  5*1  shows  some  computational  results.  We  considered  only 

completely  specified  functions  of  up  through  three  variables.  Therefore 

•3 
we  have  n  =  3  and-  m  =  2  =8. 

Total  computation  time  to  exhaust  all  optimum  networks  for  each 
of  80  switching  functions  was  about  ^k   minutes.  All  functions  were 
realized  within  6  gates.  The  result  indicates  the  computational  feasibility 
of  integer  programming  approach  and  will  encourage  its  wider  application 
to  logical  design  problems. 

As  reported  elsewhere   ,  in  case  of  NOR  gate  network  synthesis, 
a  further  significant  improvement  of  computation  time  was  achieved  by 
utilizing  intrinsic  properties  of  NOR  gates,   (i.e.  all-interconnection 
formulations  and  others. )  Although  not  attempted,  a  similar  way  of 
improvement  in  computation  time  would  be  possible  by  suitably  modifying 
the  algorithm  of  NOR -AND  combination  network  synthesis. 
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6.  Design  Of  An  Optimum  Sequentical  Network 

Synthesis  of  a  sequential  network  with  the  minimization  of  a 
certain  network  parameter  such  as  the  number  of  gates  is  known  to  be 
very  difficult  mainly  because  of  the  addition  of  state  assignment  problem. 
Here  we  will  show  that  this  problem  also  can  be  formulated  as  an  integer 
linear  programming  problem. 

When  we  design  a  sequential  network,  various  restrictions  such  as 
the  realization  by  a  shift  register  or  the  realization  by  certain  types  of 
flip-flops  are  usually  imposed  on  a  network  to  be  synthesized  for  engineering 
consideration.  As  most  of  these  restrictions  can  be  incorporated  by 
adding  extra  linear  inequalities,  only  general  formulation  will  be  discussed. 

Two  problems  will  be  considered  in  the  following.  The  first  problem 
is  the  design  of  a  sequential  network  with  the  minimum  number  of 
states.   In  other  words,  first  minimize  the  number  of  states  in  the  state 
transition  table  according  to  the  conventional  switching  theory  (The 
sequential  network  synthesis  method  which  most  of  text  books  of  switching 
theory  discuss)  and  then  the  procedure  of  Section  6.1  will  synthesize 
an  optimum  network  with  the  most  appropriate  state  assignment.  Since  the 
optimization  of  the  network  and  the  state  assignment  will  be  made  by 
integer  programming  approach  only  after  the  number  of  the  states  is 
minimized,  a  sequential  network  which  will  be  synthesized  may  not  have 
the  minimum  number  of  gates. 

Contrary  to  the  first  problem,  the  second  problem  is  the  synthesis 
of  an  optimum  sequential  network  using  the  input-output  relation  rather 
than  the  state  transition  table.  An  appropriate  state  transition  table 
and  state  assignment  such  that  the  number  of  gates  in  a  network  is 
minimized  will  be  made  simultaneously  by  integer  programming  approach. 
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6.1.  A  Sequential  Network  With  The  Minimum  Number'  Of  States 

Our  problem  is:   Given  a  state  transition  diagram  of  a  sequential 

[31C71 
network      ,  find  a  sequential  network  with  the  minimum  number  of 

building  blocks  and  a  corresponding  state  assignment.  As  shown  in  Fig.  6.1.1, 

our  sequential  network  is  assumed  to  consist  of  a  combinational  network 

and  loops  with  delays.   The  combinational  network  is  a  multiple  output 

feed-forward  network  of  building  blocks  which  has  been  discussed  in  the 

previous  sections. 


Each  s .  is  an  internal  variable 

1 

which  is  fed  back  to  the  inputs  of 
the  network  with  a  unit  time  delay. 
The  vector  s  =  fs,,  ..•,  sv) 

is  called  the  (internal)  state 

of  this  sequential  network. 

x.. ,  . . . ,  x  are  ( external)  input 

variables  and  zn,  ...,  z  (external) 

output  variables,  s.,  x.,  z,  assume 

the  value  1  or  0. 


Fig.  6.1.1 
Realization  of  a  sequential  network 


1 


*n 


C  omb  in  at  i  on  al 
Network 


-*•  z. 


V 


As  known  in  switching  theory,  the  state  transition  diagram  completely 
specifies  the  behavior  of  the  sequential  network.  The  transition  shown 
in  Fig.  6.1.2  shows  that  if  the  present  state  is  s    and  the  external 

input  vector  x  J  is  applied  to  the  network,  then  the  network  will  be 

-( 2)  Wfc) 

in  the  next  state  s   ,  emitting  the  output  vector  z   .   The  combinational 

network  in  Fig.  6.1.1  realizes  this  relation,  i.e. 


(6.1.1) 


33 


Fig.  6.1.2 
State  Transition  Diagram 


■where  f  and  h  are  vectors*  whose  coordinates  are  functions  of  s    and 

_»/  •  \  -M.  _ * 

x   .   Introducing  f  and  h,  we  have  essentially  a  multiple-output  network 
with  V+n  inputs  (the  external  and  internal  inputs  together)  and  V  +  q 


outputs.   The  difference  of  the  network  from  that  discussed  in  the 

prev 


previous  sections  is  that  x  J  and  z    are  specified  in  advance  but 


are  unknown  variables. 

An  integer  linear  programming  problem  for  designing  a  multiple- 
output  network  whose  outputs  s    are  fed  back  to  the  inputs  of  the 
network  through  the  relation  (6.1.1)  can  be  directly  formulated  with  the 
technique  developed  in  the  previous  sections. 

First  the  number  of  the  states  in  a  state  transition  diagram 
which  is  either  completely  or  incompletely  specified  is  minimized.  Let 
the  minimum  number  of  the  states  be  D,  and  find  p  such  that 

T  =  <  logg  D  >  (6.1.2) 

where  <x>  is  the  integer  closest  but  not  smaller  than  x.  Then  set 

V  to  F.   In  other  words,  consider  the  sequential  network  of  Fig.  6.1.1 

with  V  =  P  internal  variables. 


*   f  has  q  coordinates  each  of  which  is  a  function  of  (V  +  n)  variables. 
h  has  V  coordinates  each  of  which  is  a  function  of  (V  +  n)  variables. 

3U 


Assume  that  the  transition  shovm  in  Fig.  6.1.2  takes  place, 

-w  *  ^ 

in  other  words,  when  the  state  of  the  network  is  in  s    and  the 

~ii)  ~\i)  ~*(k) 

inputs  are  x   ,  the  next  state  is  s   and  the  outputs  become  z   . 

Then  consider  the  multiple  output  feed-forward  combinational  network 

with  the  n  +  r  inputs,  x    and  s    and  the  q  +  r  outputs,  z    and 

s   ,  breaking  all  the  feed-back  lines  of  Fig.  6.1.1.  Then  derive  the 

inequalities  for  this  multiple  output  network  according  to  the  procedure 

of  the  previous  sect ions (according  to  the  procedure  of  Part  I  [9],    if 

the  network  is  to  be  synthesized  with  threshold  gates  )  for  each 

transition  in  the  state  transition  diagram.   In  these  inequalities 

s    and  s    are  kept  unknown  variables  while  x    and  z    are  known 

constants. 

— H  '  \ 
The  fact  that  variables  s    are  unknown  causes  no  difficulty 

because  the  formulation  discussed  so  far  is  already  linear  in  the  input 

variables  xy'   and  the  output  variables  P^(k)  (See  (3.1.10  and  (3.1. 5)). 

When  the  state  transition  diagram  is  incompletely  specified, 

_Y  •  n     — Yv^ 
some  coordinates  of  s    and  z    are  not  specified.  Accordingly  the 

corresponding  equalities  for  (6.1.1)  are  eliminated.  Therefore  the 

resulting  multiple-output  functions,  f  and  h,  are  generally  incompletely 

specified. 

Let  us  describe  a  procedure  for  designing  a  sequential  network 

which  has  the  minimum  number  of  building  blocks   (These  blocks  constitute 

the  combinational  network  in  Fig.  6.1.1.)  under  the  conditions  that  the 

network  has  the  minimum  number  of  states  and  that  the  network  has  the 

minimum  number  of  feed-back  lines.   The  procedure  is  similar  to  that  in 

Section  k.      Let  R  be  the  number  of  generative  building  blocks  in  the 

network. 
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Our  procedure  is  as  follows: 

(1)  Set  R  =  r  (or  set  R  to  a  lower  bound  of  the  number  of  building 
blocks  which  is  found  by  some  means. ), 

(2)  Solve  the  integer  linear  programming  problem  for  a  multiple- 
output  feed-forward  combinational  network  with  R  building 
blocks  which  has  (n  +  r)  inputs  and  (r  +  q)  outputs,  optimizing 
an  objective  function.   If  it  has  a  solution,  we  have  found 

an  optimal  network.   If  not  go  to  (3). 

(3)  Increase  R  by  1  and  go  to  (2). 

If  the  same  type  of  objective  function  as  that  of  Section  k   is 
chosen,  we  can  derive  a  sequential  network  with  the  minimum  number  of 
interconnections  and  connections  among  the  networks  of  the  minimum  number 
of  building  blocks. 

Remark  6.1. 1:   Application  of  above  procedure  to  a  state  transition  diagram 
which  is  not  minimized  may  be  desirable,  because  the  increase  of  the  number 
of  states  may  reduce  the  number  of  building  blocks  in  the  network.  Note 
that  the  number  of  states  in  an  optimum  solution  by  this  approach  either 
remains  the  same  as  that  of  the  state  transition  diagram  or  is  fewer 
(i.e.  some  states  may  be  merged  in  an  optimum  solution  )  but  that  it  will 
never  be  more  than  the  number  of  states  in  the  non-minimized  state  transition 
diagram. 

Remark  6.1.2:   In  this  section  we  assumed  that  the  concrete  values  of  the 
input  vector  x    and  the  output  vector  z1   are  given  in  advance.   In 
other  words,  the  input  state  assignment  and  the  output  rtate  assignment 
are  already  determined.  However  the  above  synthesis  procedure  can  be 
easily  modified  to  the  synthesis  problem  of  a  sequential  network  where  the 

input  state  assignment  and  output  state  assignment  are  to  be  done  in 

-<±) 

addition  to  the  assignment  of  internal  variables  s   .  This  is  essentially 

logical  design  of  an  optimum  sequential  network,  assuming  only  a  state 

transition  diagram. 
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Only  modification  for  this  problem  is  to  keep  the  variables  x    and 
z1   unknown  variables  and  to  determine  their  values  during  the  network 

■41) 

synthesis  procedure  in  the  same  way  as  the  variables  s    discussed  in 
this  section. 
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6.2.  An  Optimum  Sequential  Network  Based  On  The  Input-Output  Relations 

In  this  section  we  will  discuss  a  design  procedure  which  will 
yield  a  sequential  network  with  the  minimum  number  of  building  blocks*, 
without  using  a  state  transition  diagram.  The  procedure  is  based  on 
the  given  relations  between  the  inputs  and  the  outputs  of  a  network  to 
be  synthesized  but  not  on  the  state  transition  diagram  of  the  minimum 
number  of  states.   The  number  of  building  blocks  included  in  the  network 
synthesized  by  this  procedure  will  be  absolutely  minimum. 


x{j\t) 


A  mathematical  model 
(Mealey's  model)  is  shown  in 
Fig.  6.2.1.   The  sequential 
network  consists  of  a  multiple- 
output  feed-forward  network 
consisting  of  R  building  blocks ^l  (t-l) 
and  the  R  feedback  loops  with 
unit  time  delay.   At  the  time 
t,  the  j  th  external  variables, 
x^(t),  . ..,  x^'(t),   are  supplied 


Fig.  6.2.1 
Sequ&nMal  Network 


4d)(t) 


H»-a-i"[J)(t) 


*  The  number  of  delay  elements  in  Fig.  6.1.1  is  not  counted, 
though  our  theory  can  be  extended  to  such  case. 
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to  a  feed-forward  network  in  whi^h  there  is  no  time  delay.  The  outputs 

of  same  building  blocks  appear  as  external  outputs  of  this  sequential 

network,  z^J'(t),  . ..,  z^(t).  The  outputs  of  all  R  building  blocks,* 

P;   (t),  •••j  Pi  (t),  are  fed  back  to  the  inputs  of  the  network  after 
1  ft 

one  unit  time  delay.   In  other  words,  at  the  time  t  4-  1,  the  feed- 
forward network  has  the  inputs  x!j^(t+l),  ...,  x^(t+l),  P^J'(t),  ..., 
p;a')(t)  and  the  outputs,  z$*"(t+l),  ...,  z'^(t-f-l),  P^(t+l),  ...,  p£*"(t+l). 

Consider  all  input  sequences  each  of  which  consists  of  exactly 
t  successive  external  input  vectors  and  starts  from  the  initial  state 
of  the  network,  (P£^(0),  ...,  P^(o)),  i.e.  P^(0).   Let  the  j  th 
input  sequence  be  x'^(l),  x^(2),  ...,x^(t).  Assume  that  the  output 

sequence  of  the  network  z"   (l),  z"   (2),  ...  z       (t)  corresponding  to 
this  input  sequence  is  known.  Then  consider  a  multiple -output  feed- 
forward network  with  the  feedback  loops  open,  as  shown  in  Fig.  6.2.2, 

where  the  external  inputs  are  x;^(l),  ...,  x^'(l),  P;^(0),  ...,  P^(o) 

x  n  x  ft 


and  the  outputs  are  zlj      ( l) ,    ... 

1  R      x^(t 

Then  write  linear  inequalities   /  .\ 

40)(tf- 

according  to  the  method  of  the 
previous  section  2  to  k.      (if   x^  (t) 


Fig.  6.2.2 
Multiple -Output  Network  With  Feedback 


each  building  block  is  a 
threshold  element, 


p[j)(t-l) 
p(j)(t.l)j 


zi^(t) 


*  A  vector  (P^   (t),  ...,  P^'(t))  represents  an  internal  state  of 
J.  ft 

this  sequential  network. 
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according  to  the  method  of  Part  I   .)  In  those  inequalities,  xjJ  (l),  . .., 

x  J  (l),  z..   (l),  . ..,  z   (l)  are  known  constants  and  the  initial  state* 
n     *   1   v/'    »  q 

(p|J'(0),  ...,  P^  (o))  andP^(l),  ...,  P^(l)  are  unknown  variables. 

One  unit  time  later,  we  will  have  the  feed-forward  network  where 
the  external  inputs  are  x^'(2),  F^(l)  and  the  outputs  are  z^'(2),  P^'(2). 
Write  inequalities  for  the  network  with  these  inputs  and  outputs.   In 
these  inequalities,  x^"(2),  ...,  x^'(2),  z|J'(2),  ...,  z'^'(2)  are  known 
constants  but  pj«"(l),  ...,  P^(l),  pj^(2),  ...,  P^(2)  are  unknown 
variables. 

Repeat  this  process  up  through  the  t  th  input  vector,  where 

■p 
t  =  2  +  D  -  1  and  where  D  is  the  minimum  number  of  states  in  (6.1.2) 

when  the  number  of  states  of  a  state  transition  diagram  for  a  given 

network  to  be  realized  is  minimized. 

Repeat  the  above  process  for  all  possible  2  input  sequences  of 

length  t.   For  all  sequences,  the  initial  states  F*   (0),  P   (0),  ..,,?    (0) 

are  constrained  to  be  the  same: 

p(l)(o)  =  p(2)(o)  =  ...  =  p(2T)(o). 


Design  procedure  is  as  follows.  To  begin  with,  set  R  =  P.  Solve 
the  integer  linear  program  by  minimizing  the  objective  function  which  is 
the  number  of  interconnections  and  connections,  under  the  above  inequalities. 
If  not  feasible,  increase  R  by  one  and  solve  the  linear  program  again. 
Eventually  we  will  have  a  sequential  network  of  the  minimum  number  R_ 

of  building  blocks  and  as  the  secondary  objective,  the  minimum  . 
number  of  interconnections  and  connections.  Rn  is  bounded  by 

*  We  may  assume  that  the  initial  state  P^(0),  ...,  P^  (0)  is 

constant  if  engineering  consideration  requires. 

ko 


r<^0-RD'  (6.2.1) 

where  R_  is  the  number  of  "building  blocks  synthesized  by  the  procedure 

of  Section  6.1  after  the  number  of  states  is  minimized  first. 

In  an  optimum  solution  P*   (t)  with  different  j  and  t  may 

become  identical  and  also  some  of  the  loops  may  turn  out  to  be 

open.  • 

Note  that  p|J'(t),  ...,  P^(t)  at  the  inputs  and  the  outputs 

in  Fig.  6.2.2  remain  unknown  variables  until  an  optimum  solution  of  the 
integer  linear  program  is  obtained,  while  in  the  conventional  switching 
theory  the  feed-forward  network  is  separately  synthesized  after  internal 
states(P,   (t),  ...,  P^.J'(t))'s  are  assigned  binary  variables. 

J.  K 

Setting  T  =  2+D-lis  due  to  the  theorem  known  in  autometa 

[31 
theory    that  if  two  sequential  network  with  D  and  E  states  give  the 

same  output  response  for  any  input  sequence  of  length  D  +  E  -  1,  then 

they  give  the  same  response  for  all  input  sequences  of  any  length.  For 

the  network  of  Fig.  6.2.1  the  number  of  states  is  2  . 

Since  the  numbers  of  inequalities  and  variables  are  generally 
numerous,  computation  may  be  difficult  except  for  very  simple  cases. 

In  order  to  reduce  the  numbers  of  inequalities  and  variables, 
the  above  procedure  may  be  modified  as  follows:   Consider  inequalities 
only  for  input  sequences  which  look  important  and  solve  the  integer 
program  with  them.   Check  whether  the  solution  satisfies  other  input 
sequences.   If  the  solution  does  not  satisfy  some  input  sequences  (i.e. 
the  solution  network  does  not  give  correct  response  for  these  sequences), 
add  inequalities  for  these  input  sequences  to  the  current  set  of  inequalities. 
Then  solve  the  integer  program.   Repeat  this  procedure. 

kl 


Remark  6.2.1:   The  problem  stated  in  Rer ark  6.1.2,  i.e.  the  design  of 

an  optimum  sequential  network  where  assignments  of  1  and  0  to  the  input 

_J±)  vk) 

variables  x    and  the  output  variables  z"   are  to  be  determined  can 

be  also  formulated  by  the  above  approach,  keeping  x    and  z" 
unknown  variables. 
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7.   Diagnosis  Of  A  Network  By  Irteger  Linear  Programming 

The  procedure  discussed  in  this  paper  can  be  applied  to  the  diagnosis 
of  a  given  switching  network. 

For  example,  assume  that  the  types  of  faults  of  a  given  network 

are  only  being  open-circuited.   (Interconnections  are  assumed  not  to 

be  short-circuited.)  Assign  variables  a.,  and  w    to  the  interconnections 

ik     % 

and  connections  which  exist  in  the  given  network.   (No  variables  are 

assigned  to  interconnections  and  connections  which  do  not  exist  in  the 

network. )  If  the  interconnection  from  the  i  th  building  block  to  the 

k  th  is  open  (faulty),  then  Oi       =  0  and  otherwise  (not  faulty),  Oi       =  1. 

ik  ik 

w 
w-  '  s  are  similarly  defined. 

A  diagnosis  procedure  is:  First  obtain  the  response  of  a  network 

to  be  diagnosed  for  each  input  vector.  Then  solve  an  integer  linear 

program  which  is  formulated  for  this  network  by  the  procedures  of  the 

[91 
previous  section  or  Part  I   .   Obtain  all  solutions. 

If  some  Oi       or  w  »      are  0  in  some  solutions  but  not  in  all,  the 
corresponding  interconnections  or  connections  may  be  open. (i.e.  faulty). 
If  some  cc       or.w„   are  0  in  all  solutions,  the  corresponding  inter- 
connections or  connections  are  definitely  open. 

By  interpreting  variables  differently,  more  sophisticated  diagnosis 
is  possible.  For  example,  diagnosis  to  find  malfunction  of  gates  can 
be  formulated.  Also  diagnosis  to  find  short-circuited  interconnections 
in  addition  to  open-circuited  interconnections  and  malfunction  of  gates 
can  be  formulated,  though  computation  would  be  even  more  time-consuming. 
If  we  assume  the  types  of  faults,  or  if  we  assume  a  single  fault,  the 
computation  would  be  easier.  More  sophisticated  diagnosis  will  be 
considered  elsewhere. 
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8.   Conclusion 

Some  important  logical  design  problems,  which  are  known  to  be 
very  difficult,  were  formulated  by  integer  linear  programming  approach. 
The  approach  can  be  applied  to  other  logical  design  problems  with  minor 
modifications,  though  we  did  not  discuss  them. 

Since  the  size  of  the  integer  linear  programming  problems  grows 
very  rapidly  as  the  number  of  variables  n  or  the  number  of  building 
blocks  R  increases,  the  size  of  the  problems  which  are  computationally 
feasible  would  be  limited.  However  we  showed  that  an  important  logical 
design  problem  such  as  the  design  of  an  optimum  network  with  a  mixture 
of  NOR  and  AND  was  computationally  feasible. 

The  optimality  is  guaranteed  by  our  approach  but  if  appropriate 
integer  programming  algorithms  such  as  the  implicit  enumeration  are  used, 
a  pseudo-optimum  network  may  be  obtained  if  the  computation  time  is 
limited. 

Diagnosis  of  a  given  network  can  be  made  by  slightly  modifying 
the  above  approach. 
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List  of  Symbols 

Here  listed  are  notations  used  throughout  Part  I  and  Part  II. 
Some  other  notations, which  are  used  locally,  are  not  listed  here, 
f :     function  to  be  realized  by  a  feed-forward  network. 

x   :     the  j  th  input  vector  of  f,    or  the  j  th  external  input 
vector  of  a  network.  Each  component  assumes  0  or  1. 

m:     the  number  of  specified  input  vectors. 

n:     the  dimension  of  x. 

w.:     the  weight  of  the  £   th  external  input  variable  x«  to  the 
k  th  gate. 

T  :     the  threshold  of  the  k  th  threshold  gate. 

Jjk 

oc .  :     the  weight  of  the  interconnection  from  the  i  th  threshold  gate 
to  the  k  th  threshold  gate. 

P^  :     the  output  of  the  k  th  threshold  gate  for  x^'.  The  value 
is  1  or  0. 

P:,  :     the  contribution  of  the  output  P.J'  of  the  i  th  threshold 

(  ') 
gate  to  the  input  of  the  k  th  threshold  gate  for  x   , 

i.e.  p[p   =  P)^   a.,  . 
lk     i    lk 


v  •.  assumes  1  if  x.  is  connected  to  the  k  th  building  block. 

0  otherwise . 


cp  :     assumes  1  if  the  i  th  building  block  is  connected  to  the  k  th 
building  block.   0  otherwise. 

lk         i   Yik 

k  k 

\i 0:  assumes  1  if  w.  f  0. 


;r 
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v.,  :     assumes  1  if  cp._  4  0. 
lk  lk 

K:  assumes  1  if  the  k  th  threshold  gate  (or  building  block)  is  actually- 

used,  when  a  sufficiently  large  number  of  threshold  gate  (or  building 
blocks)  is  assumed  to  be  included  in. a- network. 

f  :     assumes  1  if  the  k  th  gate  (or  building  blocks)  is  used  for 
the  t  th  function,  in  a  multiple -output  network. 

U:     a  sufficiently  large  positive  number. 

R:     the  total  number  of  gates,  or  building  blocks,  in  a  feed- 
forward network. 

g  :     the  function  of  the  given  r  th  type  building  block. 

y :     the  j  th  input  vector  to  the  k  th  generative  building  block 
in  a  network. 

cp.  (d,  k):     assumes  1  if  the  d  th  generative  building  block  is  connected 
to  the  i  th  input  of  the  k  th  generative  building  block  in 
a  network.   0  otherwise. 

v. (i,k):     assumes  1  if  x.  is  connected  to  the  i  th  input  of  the  k  th 

1  li 

generative  building  block  in  a  network.  0  otherwise. 

P   (k):     the  output  of  the  k  th  generative  building  block  in  a  network 
for  P*3'. 

Pp}(d,k):     =  PU)(d)  cp.  (d,k). 

B:     the  number  of  types  of  given  building  blocks. 

H:     the  number  of  inputs  to  a  building  block. 

C:     the  number  of  threshold  gates  necessary  to  realize  a 
generative  building  block. 

E:     the  set  of  inequalities  which  characterize  the  generative 
building  blocks  in  a  network. 
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9   (k):     assumes  1  if  the  r  th  building  "block  is  selected  for  the 


r 


k  th  generative  building  block.   0  otherwise. 


z       :  the  j  th  output  vector  of  a  sequential  network. 

q:     the  dimension  of  z        . 
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Appendix  A 

Expression  of  g_,  . ..,  g_  "by  a  Multi-Threshold  Threshold  Gate 

A  mult i -thresh old  threshold  gate  is  a  threshold  gate  with  more  than 
one  threshold1         .  The  output  of  the  gate  is  appropriately  defined 
by  many  ranges  of  the  value  of  Sw  .y.   separated  by  thresholds. 

A  single  mult i -threshold  threshold  gate  can  be  used  as  a  basis  for 

the  integer  linear  inequality  expression  of  g  ,  ...,  g^.  Sometimes  this 

results  in  a  more  concise  expression  than  that  of  Section  2. 

Introducing  a  set  of  weights*,  w_ ,  w ,    ...,  mi     and  a  set  of  thresholds* 

L        d  n 

Tx,  T2,  . . . ,  TQ  where 


Tx  <  T2  <  . . .  <  TG  , 


(Al) 


we  want  to  express  the  given  functions,  g  ,  ...,  g^  collectively  such  that 

Fig.  Al  Separation  of 
values  of  gx,  ...,  ^ 


the  following  two  conditions  are  satisfied: 


H 


Xj) 


(i)  The  values  of  Z  w.  y.j;  for 

11 
i=l 

all  different  combinations  of  1  and 


0  of  y 


(J) 

_      ,    . . . 


,(d) 


'H 


are 


partitioned  into  the  following 
G  +  1  ranges  by  the  thresholds. 


Range 

Value  of 

2  w.y:^ 
11 

gl 

H 

•  •  • 

% 

0  th 

-  «  to  T  -1 

0 

1 

0 

1  st 

T     to  T2-l 

1 

1 

1 

2  nd 

T2  to  T  -1 

0 

0 

1 

•   •   •   • 

G  th 

T^,  to  oo 

•   •   • 

0 

1 

0 

*  The  values  of  weights  and  thresholds  are  assumed  to  be  integers 
without  loss  of  generality. 
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0  the  range: 


1  st  range: 


H 

Z    w.   y)u/  <    T,   -  1 
.=1     i     i       -       1 


,(J) 


T     <    Z  w.   y.^  <  T.  -  1 

1  —  li       —     2 


(G  -  1)    st  range:      TG-1  -    z  *±  y^       <    TG  "  x 


G  th  range 


T^  <    Z  w.  y:^ 

G  —  li 


(A  2) 


(ii)      In  Fig.    Al,    if  an  input  vector  y  which  belongs  to 


a  range  is 


a  true  vector  of  g  ,  then  1  is  filled  in  the  entry  for  this  range  and  g  , 
and  0  otherwise.   Then  for  each  g  ,  each  range  should  contain  either 
only  true  vectors  of  g  or  only  false  vectors  of  g  ,  so  that  the  output 
value  of  g  is  definable  for  each  range. 

Weights  and  thresholds  which  satisfy  the  above  conditions  (i)  and 
(ii)  always  exist*.   However  those  with  the  minimum  number  of  thresholds 
would  give  a  consise  expression  of  inequalities.  We  can  obtain  such 
weights  and  thresholds  by  using  an  integer  linear  programming  approach, 
as  will  be  discussed  in  Appendix  B. 

Assume  that  we  have  obtained  the  set  of  weights  and  thresholds 

which  satisfy  the  above  conditions,  (i)  and  (ii).   Let  us  express  the 

ranges  of  Z  w.  y.    in  inequalities  as  follows,  introducing  integer 

variables  o    which  assumes  1  if  the  value  of  Z  w.  y.   is  in  the  s  th 
s  11 

range  and  0  otherwise. 


i-1  (i) 

*  If  we  set  w.  =  2    ,  i  =  1,  2,  . . . ,  H,  then  the  sum  Z  w.  y.  ' 
i        '      7777  ±  o  1 

for  each  y     assumes  a  different  value  and  the  values  of  each  g 

are  separated  by  thresholds  which  are  set  to  the  different  values  of 

Z  w.  y.   .   Thus  the  conditions  (i)  and  (ii)  are  satisfied, 
ii 
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Ad)      Ad)  ,  Ad)  _\ 

O^    t  O,    +••••+  Op    -  J- 


(A  3) 


H 

E 

i=l 


-  E  w.  y)^   >  -  Tn  +  1  -  U  (1-  b\     )         ^ 
.,ii—    1  0 


E  w.  y{3)   >  T  _  u  (!  _  s(j)) 
li—l  1 


s  w.  y:^  >-t0  +  i-u(i-  6;J0 

11—2  1 


E  w.  yp}  >  T2  -  U  (1  -  6^) 


E  w.  y:^  >  -  T0  +  1  -  U  (l  -  6^) 
i  i   —    3  2 


> 


E  w.  y!0'^  :  T  -  u  (l  -  6^0 

11    —    U  Lr 


(All) 


y 


(  j  =  1,  2,    ... ,   m  )# 


These  inequalities  implies  that  if  E  w.  y.   assumes  a  value  in  the 


s  th  range,  b^     =   1  and  5^J;  =  0  for  t  |=s.   Then  from  the  inequalities 


(j) 

Jt 


C(J) 


(A3)    and  (Ai+),    g     (y         )    can  be  expressed  as 


k€S(r) 


(A  5) 
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where  S(r)  is  the  set  of  ranges  which  have  entries  1  in  the  column 

of  g  in  Fig.  A  1.   The  right  hand  side  of  (A  5)  can  he  used  instead 

of  Q  J  of  Section  2.1  for  the  selection  of  an  appropriate  building 

r 
block  discussed  in  Section  2.2. 

Consequently  (A5)  together  with  (A3)  and  (Ak) ,   for  all  r  =  1,  2,    ...,   B, 

express  the  given  building  blocks  g  ,  ...,    g^. 

Example  A.  1 

Consider  the  two  building  blocks  discussed  in  Example  2.1. 

gl  =  yi©y2 

op   <y  *i  <y  p  * 

An  example  of  weights  and  threshold  which  separate  the  values  of  each 
function  is 


w  =  w2  =  1 
T1  =  l 
T2  =  2  . 

This  set  of  w„ ,  w«.  T.,  and  T^  partitions  the  values  of  Z  w.  y.    into  the 
1'   2'   1      2  *  1  Ji 

following  three  ranges: 

the  0  th  range,     E  w.  y.   <  0 


■.  y:J''  <  1 


the  1  st  range,   1  <  Ew.  y. 


the  2  nd  range,   2  <  w.  y.  '  . 
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Namely,  each  input  vector,  y    ,  actual  value  of .which  is  determined 
during  the  computation  of  the  whole  integer  programming  problem,  belongs 
to  a  range  as  follows 


the  0  th  range 
the  1  st  range 
the  2  nd  range 


(00) 
(01),  (10) 

(11). 


This  gives  the  following  inequalities  corresponding  to  (A. 3)  and  (A.k)    ; 


A  3)    ,  Aj)    ,  AS)   _  -, 
B0   +  81   +  &2   -  1 , 


(  J  =  1>  2,  ...,  m ) 


(A.6) 


>0  -  U  (l-  6^) 


P(d)  m  VU) 

1     y2 


\ 


0 


y^}  +yp}  >i  -  u  (l-  8^}) 
_y<J>  .yp)>..1..u(i-6p)) 

yp}  +  y^}  >  2-  u  (l-  8^}), 


(A.T) 


y 


(  j  =  1,  2,  ...,  m ). 


As  easily  seen,  for  each  range,  the  values  of  g  and  g  are  assigned 
as  follows: 


Range 

gl 

&2 

0 
1 
2 

0 
1 
0 

0 
0 

1 

(A.8) 
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In  other  words  s(l)  =  [1),   s(2)  =  12), 

8l  (?3))  =  4d) 


(A.  9) 


52  &$)    =  B^ 


Consequently  (A. 9)  together  with  the  inequalities  (A. 6)  and  (A. 7) 
express  g  and  g  . 

The  second  example  is 


00  ■"  j-i   ./p. 


This  is  written  as 


h&*h  =4J)  +4i}- 


using  the  same  weights  and  thresholds  as  the  first  example. 
Remark  A.  1 

If  g  ,  . . . ,  gp  are  all  symmetric  in  variables,  we  can  always  set 
the  weights  as  wn  =  w0  =  . . .  =  w„  =  1.  This  may  simplify  the  process 
and  usually  give  a  considerably  concise  expression. 
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Appendix  B 

Minimum  Realization  of  A  Multi -Thresh old  Threshold  Gate  For  A  Generative 

Building  Block 

In  Appendix  A,  we  discussed  representation  of  a  generative  building 

block  with  a  multi -thresh old  threshold  gate.  Here  we  will  discuss  a  design 

procedure  of  an  optimum  set  of  weights  and  threshold  by  integer  programming 

approach  such  that  the  number  of  thresholds  is  minimized  first  and  the  sum 

H 
of   E  |w.|  is  minimized  second.  The  gate  designed  here  will  be  used  to 
i=l   X 

express  a  generative  building  block  for  g_,  g_,  . ..,  g„  in  Appendix  A. 

Let  us  introduce  zero-one  variables  €  ,  r  =  1,  2,  ....  B  and 

rs 

s  =  0,  1,  ...,  G*   £   assumes  1  if  and  only  if  the  value  of  g  in 
the  s  th  range  of  Fig.  A  1  is  1.   Otherwise  e   =0.  Note  that  the 

value  of  e   is  independent  of  input  vector  y 

rs 

Next  let  us  introduce  zero-one  variables  n   : 

rs   ' 

n(d)   =  e       &(j),  (Bl) 

rs  rs     s     '  v      ' 

TT 

r       —       A~y       Ct  y         •    •    o    y        -D  y  S       —       \J  y        d-y         .    .    •  y         \J  y  J       =       S.y       cL  y         .   .   »  y        ct        . 

Then  consider  the  following  inequalities; 

G   /.v 

E  7tU;  >  1  (B2) 

^  rs  —  v  ' 

s=0 

for  each  true  vector  y        of  g.  and 


G   /  .x 

2  nKD)  <  0  (B3) 


rs  — 
s=0 
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for  each  false  vector  y  ^  of  g  .   (B2)  implies  that  for. each  true 
vector  y     of  g  ,  there  exists  a  range  (the  s.th)  -which  satisfied 

5    =  1  and  e       =1.   (B  3)  implies  that  for  each  false  vector  y  ^ 

s  rs 

of  g  ,    any  range  (the  s  th)  with  S^J'  =  1  satisfies  e   =0.  This  is 

JL  S  I*S 

the  property  which  should  he  satisfied  by  a  multi-threshold  threshold 
gate  to  he  designed. 

The  non-linear  relation  (Bl)  can  he  expressed  in  linear 
form  as  follows: 

_g  (J)  +  itd)   +  l  >  0 


-  € 

rs    s      rs 


6   +  5^  -  2*^'  >   0  > 

rs     s      rs     — 


(BU) 


(r  =  1,  2,  ...,  B);(  s  =  0,  1,  ...,    G);(j  =  1,  2,    ...,   2H  ). 

Consequently  (B2)  for  each  true  vector  y     of  g  and  (B3) 

for  each  false  vector  of  g  ,  r  =  1,  2,    . ..,  B,  together  with  the 

inequalities  (Al),  (A3),  (A^)  and  (Bh) ,    characterize  completely  a 

mult i- thresh old  threshold  gate  with  the  property  required  for  g_ ,  ...,   g_. 

l  a 

A  procedure  for  obtaining  an  optimum  structure  is  to  solve  these 

inequalities  with  the  objective  function 

H 

S   |w,  |  ,  (B5) 

i=l 


starting  from  G  =  1  and  increasing  G  by  one  each  time  until  we  get  a 

feasible  solution.  The  procedure  is  sklar  to  those  discussed  in  the  text. 

However,  note  that  variables  w  .  and  T  may  assume  negative  values. 

1      s 

56 


By  putting  each  variable  as 


+ 
w.   =  w.        -  w. 
11  1 


T     =  T     +  -  T     ",  (B.6) 


s  s  s 

we  can  consider  the  problem  as  a  integer  programming  problem. 

(B.5)  is  actually  written  as 

H    , 

S  (w~r+w).  (B.7) 

i=l 

Also  note  that  each  y  J  is  considered  as  constant  vector  ranging  from 

it 
(00  ...  0)  to  (l  1  ...  l)  corresponding  to  j  =  1,  2,  ...,  2  ,  whereas 

others  such  as  w,  T,  8   ,  e  ,  a         are  all  unknown  variables. 

'      '      s  '     rs7   rs 
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Appendix  C 

Representation  of  g  ,  . ..,  gn  by  Enclosing  the  Sets,  of  True  Vectors  and  False 
Vectors  by  Different  Sets  of  Inequalities 


Fig.  C  1  Enclosure  of 
true  vectors  by  inequalities. 


Here  we  will  discuss  a  method  to  represent  g  ,    ...,   g^  by  enclosing 
the  set  of  all  true  vectors  and  all  false  veetors  of  e^ch  g^. 
by  two  different  sets  of  inequalities.  The  method  is  based  on  the 
observation  that  we  can  always  construct  a  set  of  inequalities  which 
enclose  all  true  (or  false)  vectors  but  not  false  (or  true)  vectors. 
As  shown  in  Fig.  C  1,  only  true  vectors 
denoted  by  circles  are  enclosed  inside 
the  hyperplanes  denoted  by  lines  which 
correspond  to  inequalities.   False 
vectors  denoted  by  dots  are 
outside  the  convex  set  which  is 
bordered  by  the  hyperplanes. 

These  inequalities  can  be 
derived,  for  example,  by  the  following 
procedure.   Let  P  be  a  set  of  binary 

vectors  which  we  want  to  enclose  by  inequalities.   In  other  words  P  is 
inside  the  solution  region  of  these  inequalities.   P  is  the  set  of  binary 
vectors  which  do  not  belong  to  P  and  which  we  want  to  keep  outside 
the  solution  region  of  the  inequalities.   For  each  p  e  P  form 
inequality 


°i   yi  +  a2 


y2  +  . . .  +  ah   yH  >  b 


Ui) 
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where 


a. 

1 


-  1   if  p.  =  1 
i 


1   if  p.  =  0 
i 


and 


H 


+  1 


As  can  be  easily  seen,  any  binary  vector  of  P  or  P,  except  p  ,    satisfied 
the  inequality  (C  l).   Therefore  the  set  of  inequalities  for  all  p  €  P 
is  the  inequalities  which  we  wanted. 

There  may  be  other  procedures  which  lead  to  a  fewer  inequalities. 
In  the  following,  assume  that  we  have  obtained  inequalities  whose  solution 
region  contains  a  given  set  P. 

First  consider  the  set  of  true  vectors  for  g  .  Assume  that  we 
have  obtained  inequalities  whose  solution  region  contains  the  set  of 
true  vectors  for  g.. : 


ail  yi  +  a12  y2  +  ••••  +  al  H  yH  ^  bl 


aKL  yi  +  aK2  y2  +  ••••  +aKH  yH^bK   J 
in  other  words,  y     is  a  true  vector  of  g_  if  and  only  if  y     is 
a  feasible  solution  of  (C2).   Similarly  the  set  of  false  vectors  is 
contained  in  the  solution  region  of 

Cliyi+C12  y2  +  *••    +ClHyH^(1L  1 

CLiyi+CL2  y2  +  '~    +CLHyH^dl.  J 


(C.2) 


(C3) 
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Then  from  (C  2)   and  (C  3)>    derive  inequalities; 


-ll*!0  ■ 


(d) 

12  "2 


+  ...   +  *m  y£j)  >bx  -  U  (1-  gx(y  (J))) 


aKiyi       +  ^2  Y2        + 


ciiyi     +ci2y2     + 


U) 


,(d) 


CL1  yi     +  CI2  y2        + 


rU) 


c(d)- 


+  aKHyH       >bK  -U  (1-  gl(y  vjy)) 


,<d) 


■*  (d) 


+  c]HyH      ^"DsL(y         ] 


+  c 


,(d) 


,7  (J) 


IH7^>a1-UsL(y^), 


J 


(c.U) 


(j  =  1,  2,    . . . ,  m) . 

The   inequalities    (C  h)    completely  describes  g    .      If  g      (y     ^    )    =  1, 
then   (C  k)    is  equivalent  to   (C  2)    and  if  g     (y         )    =  0,    equivalent  to 
(C  3). 

Construct   inequalities  for  each  of  gu,    g„,    • ..,    g     in  a  similar 
way.      All  these   inequalities  describe  g  ,    ...,    g   . 

i.  D 

The  selection  of  one  building  block  out  of  g  ,  . . . ,    g  can  be 

1         B 

done  in  the  same  way  as  shown  in  Section  2.2. 

The  above  approach  to  the  representation  of  g  ,  . . . ,  g^  has  a  merit 

that  we  do  not  need  to  introduce  new  variables  such  as  Q    for 

r 

threshold  gates  in  Section  2.1  and  5    of  Appendix  A  which  were  necessary 
to  describe  the  relation  in  linear  inequalities.   Since  the  computation 
time  of  most  of  currently  available  integer  programming  algorithms  appear 
to  depend  on  the  number  of  variables,  this  approach  may  be  preferable 
in  some  cases. 
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Example  C.  1 

Consider  the  following  two  building  blocks: 


denotes  false  vectors  of  g 


O  denotes  true  vectors  of  g 


1 


yi  +  y2  =  1 


Fig.  C.2    Input  vectors  and 
hyperplanes  for  g  . 


Input  vectors  for  g  are  shown  in  Fig.  C.2.   For  each  input  vector,  g 


and  g-   satisfy  the  following: 


&L  (   H  ) 

=  gl  (  00  )   =  0 

g-L   (    10   ) 

=  g;L  (  01  )  =  1, 

gg    (    00   ) 

=  g2   (   01  )   =  g2   (   10  ) 

gg    (    11   ) 

=  1  . 

=  0 


(C.5) 


(C.6) 


From  Fig.  C.2  it  is  easily  seen  that  the  true  vector  of  g  are  contained 
in  the  solution  region  of  the  following  two  inequalities: 


y^   +  y<J)  >  i 


-»  y 


(d) 


^]> 


(C7) 


-  l  , 


(J  =  1,  2, 


*). 
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